Papers with pretrained transformers
Promoting Graph Awareness in Linearized Graph-to-Text Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent applications of pretrained transformers to linearizations of graph inputs yield stateof-the-art results on graph-to-text tasks. |
| Approach: | They propose to use pretrained transformers to encode local graph structures . they find they can improve the quality of models' implicit graph encodings . |
| Outcome: | The proposed models can encode local graph structures and reconstruct corrupted inputs. |
Improving Neural Topic Models using Knowledge Distillation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current paradigms for transfer learning use general knowledge as a foundation for more specialized endeavors. |
| Approach: | They propose to combine probabilistic topic models and pretrained transformers to improve topic quality by using knowledge distillation. |
| Outcome: | The proposed framework improves topic quality over all estimated topics and in head-to-head comparisons of aligned topics. |
How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Task-agnostic data augmentations have proven widely effective in computer vision, even on pretrained models. |
| Approach: | They examine the effects of two types of task-agnostic data augmentation on pretrained transformers using 5 classification tasks and 6 datasets. |
| Outcome: | The proposed techniques improve performance on 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers. |